Adaptive Depth Computational Policies for Efficient Visual Tracking

Though convolutional neural networks have made significant improvements to the task of video tracking, they come at the cost of being extremely computationally expensive. In this work, we make the observation that different frames in a video can require different levels of network complexity in order to track with high accuracy. To exploit this, we propose a fully convolutional Siamese network that performs video tracking at multiple network depths. We learn an adaptive policy, in the form of gating functions, to control how deep to evaluate the network during runtime. Training proceeds in two phases, a finetuning phase where we train the convolutional weights to extract meaningful features for metric learning and a gating phase where we train the gate weights to balance accuracy and computational cost. Our results show that our network can achieve accuracy that is competitive with the state-of-the-art on the VOT2016 benchmark. Furthermore, we show that using conditional computation with the adaptive policy, we achieve higher accuracy than fixed-depth policies with less computational cost. The framework we present extends to other tasks that use convolutional neural networks and enables trading speed for accuracy at runtime.